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Method and System ^Teleconferencing 

Field of Invention 

The present invention relates generally to telecommunications, and more 
5 specifically, to a system and method of teleconferencing. 



applications 



Background of the Invention 

Teleconferencing systems allow 
they were in the same room. In spite of the 

1 0 these systems, they are commonly used for 
resulting reductions In travel time and cost, 
not generally be rationalized in other 
so teleconferencing is not common in these 
Traditional teleconferencing systems 

1 5 monophonic speaker arrangements at each 
teleconference, and the methodology was tc 
participants, blocking the remainder of the 
evolving and systems are now available 
signals of the participants and stereo sound. 

20 present even greater demands on the carrie 
and lower latency, which results in even hij 
explains the limited availability and use of 
Generally, each teleconferencing 
specific communication network. Presently, 

25 dominant: the public switched telephone 
These systems are typically composed 
personal computers, an access network sucji 
link, and a backbone network such as the 
or the intercity data networks. Although the 

30 greatly, the backbone networks must handle 
operate reliably and efficiently. Therefore, 
focussed on the provision of single services 
incentive for telephone companies to offer 



peo Die at different locations to converse as if 
currently high cost and complexity of 
business applications, because of the 
However, the cost and complexity can 

such as academia and private use, 
{areas. 

consisted of single microphone and 
physical location participating in the 
broadcast the loudest voice to all other 
voices. However, the art has been 
offer such added features as video 
Generally though, these new features 
networks in terms of higher bandwidth 
cost and complexity. This largely 
advanced teleconferencing systems, 
is designed to be used with a 
two communication networks are 

for voice, and the Internet for data, 
equipment such as telephones or 
as a telephony local loop or a radio 

switched telephone network (PSTN) 
leeds of users at the terminals vary 
highly standardized loads in order to 
traditional communication networks 

•ather than differentiation. There is no 
varied features or to serve small niche 
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markets as the revenues would not offset the substantial cost of developing and 
implementing these additional products. 

In voice telephony, services are implemented by having large computer 
programs running on centralized switches whi :h interrogate local and distributed 
5 databases. The local databases specify which features are enabled on a given line, 
the switch software interprets these feature lis :s and Implements the switch 
behaviour, and the switch software also interrogates the distributed databases via 
Common Channel Signajling^System No. 7 (S 57) queries. SS7 is a global standard 
for telecommunications that defines the procei lures and protocol by which network 

1 0 elements in the public switched telephone network (PSTN) intercommunicate control 
messages for basic call setup, management, stnd tear down, as well as for special 
intelligent or database services such as local r umber portability (LNP), toll-free 
(800/888) services and call forwarding. 

In PSTN, a user only has access to set vices provided by the local exchange 

1 5 carrier, which in turn may only function within t ie bounds of the SS7 protocol. 

Therefore, users can only access the switches in a limited way, and new features can 
not be added by outside parties. 

Telephony features, such as teleconferencing, may only be implemented by 
adding code to the programs running the switcies or by adding specialized hardware 

20 to the telephony network. The features available to particular users are defined in the 
local databases accessed by the switch softwa re, and adding a new type of feature 
may involve changing these databases together with the switch software that uses 
them, and may also involve purchasing and installing new types of hardware in the 
network. 

25 This limits the speed with which new fe atures can be introduced since new 

hardware and software must be designed, test 3d, manufactured and deployed. The 
inflexible assignment of tasks also makes it im >ossible to share loads between 
different types of hardware, for example to use idle tone-decoding hardware to help 
with an overload of voice-conferencing or to provision a new teleconferencing 

30 feature. 

A traditional PSTN teleconferencing system provides each user with a 
bidirectional audio communication link with eac h of a plurality of remote transceivers. 
Typically, the system includes a microphone a each location for producing an audio 
signal from that location and a transport network such as the public switched 
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telephone network (PSTN) which deliveriei each voice signal to a conference bridge 
This conference bridge mixes the voice signals and returns them to audio amplifiers 
and speakers at each location. 

The conference bridge is implemented as a new hardware component 
connected to the switch providing the service. Adding a new feature such as Dolby 
noise reduction or bass boosting requires ci physical change to the hardware and/or* 
software in every switch that offers the service. 

Changes to existing telecommunication networks are therefore very 
complicated to make. There is a rigid mod ;l and hardware structure is difficult to 
extend, so existing telephone companies a e forced to focus on broad services. 
When they do develop new products they Inevitably take a long time to bring to 
market and are expensive to implement. 

Telecommunications systems need to process the data flowing through in 
complex ways, often with processing occur ing on computer systems separated both 
geographically and administratively. Many communications paths are simultaneously 
active, and the processing applied to the various flows of data changes frequently 
and in a wide variety of ways. The softwaro needed to control these computer 
systems is generally large, complex and dif Icult to change. 

The complexity of present telecomnr unications systems software, and the 
extensive interactions between its software components, makes the development of 
new features very difficult. As well, teleconr munication services have traditionally 
been provided by large monopolies who employed proprietary equipment that only 
they had access to. Large telephone comp anies hesitate to allow open access to the 
control of their switches and servers due to the risk of failures and the resulting 
damages that would occur; therefore, only very limited access is allowed. 

Software development for telephone companiesis therefore limited to a 
"closed" group of trusted developers, which reduces the talent pool available and 
shuts out developers with new ideas for niche markets. 

In summary, problems with the PSTN include: 

1 . system complexity results in long time to bring new products to market; 

2. cost of services results in focus on fow specific services rather than diversity 
and niche markets; 
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3. existing services are provided by dec icated hardware and software which are 
inflexible and must be physically, anc) often manually, modified to offer new 
services or features; and 

4. only proprietary access to switches ajnd their software code is allowed. 
The implementation of software applications in an Internet environment is 

generally done by the software running at tho endpoints, and the IP (internet 
Protocol) network is treated merely as a con iuit for transfer of data packets between 
the two points. The routers in the IP networl; merely index internal routing tables 
using the address of data packets so that thoy know how to forward them, and do not 
generate data for either of the endpoints, or lead to instructions from either of the 
endpoints. The Internet itself may be envisic ned as a series of routers 
interconnected by an Internet backbone network designed for high-speed transport of 
large amounts of data. Users may access the Internet using personal computers in a 
number of manners including modems connected to the Public Switched Telephone 
Network (PSTN), or set top boxes connectec to existing telephone or television cable 
networks. 

Communications over the Internet can be administered using various 
protocols, over a variety of physical transfer media. A protocol is a set of conventions 
or rules that govern transfer of data between hardware devices. The simplest 
protocols define only a hardware configuration while more complex protocols define 
data formats, error detection and correction t echniques and software structures. 

The key advantages of a protocol like IP are that it allows a large network to 
function efficiently and that it offers a standardized means by which applications 
software can use that network. The main disadvantages are: 

1 . that it does not allow processing to be | 

2. that it does not allow quality of service i 1 
For example, the Internet generally will not let a user run an applet on a node 

or server. This limitation is due to the architecture of the Internet which is based on 
the international OSI (Open Systems Interconnection) standard. The OSI standard 
describes communication systems using a seven layer model, each layer being 
operable to perform certain functions. Although OSI is not always strictly adhered to 
in terms of keeping related functions together in a well-defined layer, most 
telecommunication products make an attemp to place themselves in relation to the 
OSI model. The OSI standard is not likely to change dramatically, nor is the 
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Internets use of the standard, so the Internet will not likely become an active 
component in the provision of telecommunication services. 

More importantly, the Internet does n Dt allow quality of service to be specified 
Internet communications generally rely on tha transport of data packets over various 
heterogenous networks, so even though certain links may have predictable data 
rates, for example, a privately owned T1 line total end to end transfer rate is still not 
predictable or dependable. 

Some protocols such as resource reservation protocol (RSVP) set tags and 
priorities which can influence the routers on stn Internet path a little, but not a great 
deal. The RSVP is an extension to IP that pormfts specification of quality of service 
at a technical level, in terms of parameters such as data rates and latencies. It has 
had limited acceptance due to the complexity i 
need for their switching hardware to be upda < 
unless all switches in the end to end connect on are responsive to the protocol, which 
is not generally the case. 

Therefore, typical software applicatior s operating over the Internet, such as 
teleconferencing, look at the Internet as simp y a transport network without any 
processing capability and all functionality is p aced at the participants locations. 
Implementations of teleconferencing over Internet, for example, have software at 



20 each user's personal computer (PC) that acts 



it adds to backbone networks and the 
:ed. As well, little is accomplished 



as the interface with the user, 



converting voice to data packets for IP transmission to each of the other participants 
in the teleconference. Accordingly, the user's PC also receives streams of voice data 
from each of the other participants in the teleconference and plays them through a 
sound card. 

25 This implementation suffers from sevelre scalability problems. For example, if 

there are ten participants in a teleconference, 

sufficient bandwidth to download nine simultaneous voice data streams from the 
other participants, in real time. As the bandw dth to each user would increase linearly 
with the number of participants, and the load 1 
30 of the number of participants, there would be 

resources. Clearly, this is impractical for teleconferences with a large number of 
parties or sen/ices which themselves require high bandwidth such as video or high 
quality voice. Even if the bandwidth could be obtained, there is no way to ensure that 



>n the network increase with the square 
an immense load on the network 
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ng teleconferencing system is designed 



it is consistently available, as there is no way to specify quality of service (QoS) in 
Internet applications. 

As noted above, typically, each existii 
to operate over a particular network and is not capable of cooperating with the many 
varied networks now available. These networks include public switched telephone 
network (PSTN), Internet, cellular telephone systems, satellite communications, local 
area networks (LANs) and wide area networks (WANs). Within these networks there 
are a variety of media including optical fibre, wireless or hardwired electrical 
connections, which execute communications over these networks in analogue or 
digital format using a variety of different protocols. Many of these networks have 
been widely implemented, at considerable capital cost, so it is unlikely that they will 
be quickly abandoned and a new, standard, vorld-wide telecommunications network 
constructed. Therefore, there is a need for e system which is capable of 
implementing teleconferencing over a mixed combination of communications 
networks. 

Asynchronous Transfer Mode (ATM) networks, for example, use standard 
protocols for addressing packets of data and setting up connections, and have 
typically been deployed in the core of backbone networks because of the high 
speeds at which ATM equipment operates. Elecause ATM routers are not directly 
accessible and because of the complexity of their mechanisms for describing QoS, 
these mechanisms have not been used by applications software. 

Besides the IP and ATM networks me ntioned above, there are other data 
networks such as Frame Relay and Ethernet As well, the PSTN may also be used to 
carry data, for example using trellis coding which maps digital data onto an analogue 
signal commonly; which is commonly used by Personal Computer modems. Variants 
are also evolving of each major type of network, and engineering differences 
between implementations of these networks t esutt in different performance. The 
complexity induced by this variety makes it difficult for users and application software 
to exploit all the networks available, and to e> plort any to its fullest extent 

Feature development is already diffici It for the simple application of 
teleconferencing over voice networks. As new media such as videophone, typed 
messaging, shared files and whiteboards are mixed with traditional teleconferencing 
products, and new applications such as distance learning, Internet Relay Chat and 
Internet gaming, develop, the problem is becoming even more severe. This problem 
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will grow even greater as expectations 
mapped into another, as when customers 
apply in videoconferencing or Internet 

Furthermore, even for a single app 
needs, for example, requiring different deg 
there is a need for a system which can 
becoming complex, slow to develop and 

There is therefore a need for a 
may be implemented over mixed 
complexity of such existing networks to 
architecture. 



develop for features from one domain to be 
sxpect a feature similar to call-waiting to 
garring. 

ication, different users may have different 
degrees or forms of encryption; Therefore, 
allcjw many cases and features without 

slpw in operation, 
method and system of teleconferencing that 
telecommunications networks, and addresses the 
provide an open, scalable and flexible 



Summary of the Invention 

It is therefore an object of the invention to provide a method and system of 
teleconferencing which addresses the problems described above, at least in part. 

One aspect of the invention is broadly defined as a system for 
teleconferencing comprising: three or more user terminals, each having an audio 
input and an audio output; a telecommunications network interconnecting the user 



terminals and operable to transport data tc 
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and from the user terminals; separate 



modular mixing software for each respective user terminal, executing on the 
telecommunications network, and operable: to receive separate audio signals from 
the audio outputs of the others of the user terminals; and to combine the separate 
audio signals into a signal for the audio input of the respective user terminal which 
correlates to the needs of the respective user terminal. 

Another aspect of the invention is c efined as: a server for teleconferencing 
comprising: means for interconnecting user terminals and transporting data to and 
from the user terminals; means for executing separate modular mixing software for 
each respective user terminal, the separata modular mixing software including: 
means for receiving separate audio signals from the audio outputs of the others of the 
user terminals; and means for combining tie separate audio signals into a signal for 
the audio input of the respective user terminal which correlates to the needs of the 
respective user terminal. 

An additional aspect of the invention is defined as: a method of 
teleconferencing comprising the steps of: receiving, at a separate modular mixer 
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representing a respective one of three or more 
telecommunications network, separate audio signals from audio outputs of the others 
of the user terminals; and combining the separate audio signals into a signal for an 
audio input of the respective user terminal which correlates to the needs of the 
respective user terminal. 

A further aspect of the invention is defined as: a computer data signal 
embodied in a carrier wave, said computer data signal comprising a set of machine 
executable code being executable by a computer to perform the steps of: receiving, 
at a separate modular mixer representing a respective one of three or more user 
terminals and executing on a telecommunications network, separate audio signals 
from audio outputs of the others of the user terminals; and combining the separate 
audio signals into a signal for an audio input of the respective user terminal which 
correlates to the needs of the respective user germinal. 

A still further aspect of the invention is < 
storage medium storing a set of machine executable code, the set of machine 
executable code being executable by a computer server to perform the steps of: 
receiving, at a separate modular mixer represe nting a respective one of three or 
more user terminals and executing on a telecommunications network, separate audio 
signals from audio outputs of the others of the 
separate audio signals into a signal for an aud o input of the respective user terminal 
which correlates to the needs of the respective user terminal. 
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user terminals and executing on a 



iefined as: a computer readable 



will become more apparent from the 
to the appended drawings in which: 
system in a broad manner 



Brief Description of the Drawings 

These and other features of the invention 
following description in which reference is r 
Figure 1 presents a physical layout of a teleconferencing 

of the invention; 

Figure 2 presents an exemplary physical layoiit of a teleconference system in a 

preferred embodiment of the invention; 
Figure 3 presents a block diagram of exemplary 

preferred embodiment of the invention 
Figure 4 presents a block diagram of an exerrjplary 

a preferred embodiment of the invention 



signal processing software in a 
and 

operating system architecture in 
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ments of the Invention 



Detailed Description of Preferred Embod 

A system which addresses the objecs outlined above, is presented as a 
physical layout in Figure 1. This figure presents a system 10 for teleconferencing 
between three or more user terminals 12, 1' , 16, each having an audio input and an 
audio output. The phrase terminal' is used generally in the art to describe any 
suitable manner of user audio input and output device including telephones, cellular ~ 
telephones, and personal computers with microphones and speakers or headsets. 
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ctions between the terminal and the 
audio interface between the user and the 



The audio input and output refer to the conn 
telecommunications network, and not to the 
10 terminal. 

A telecommunications network 1B intsrconnects these user terminals 12, 14 , 
16 and has the necessary functionality to transport data packets between them. The 
telecommunications network 18 also execute is separate modular mixing software for 
each respective user terminal 12, 14 , 16. Tiese separate mixers 20, 22, 24 are 

1 5 operable to receive separate audio signals ft om each of the other user terminals and 
to combine those separate audio signals intc one signal. This one mixed signal is 
transported to the audio input of the respective user terminal, in a manner that 
correlates with the needs of that respective i ser terminal. 

That is, if there were three participant in the teleconference, there would be 

20 three mixers, a first mixer for participant A, v* hich mixes the audio output signals of 
participants B and C, a second mixer for pari icipant B, which mixes A and C, and a 
third mixer for participant C, which mixes A and B. 

The use of individual mixers 20, 22, 24 executing on the telecommunications 
network 18 addresses a number of the probl ems noted above. 

25 Firstly, having individual mixers 20, 21, 24 allows each mixer to be dedicated 

to its own user and to be tailored to the limitations of the user's resources and the 
resources of the network and network connexion that services that user. For 
example, the mixer of a user having a high bandwidth connection may provide digital 
quality stereo to its user, with balanced mixing of ail participant's voices. Another 

30 user connection via an analogue, monophon c, PSTN connection may send only the 
strongest voice to its user, blocking voice signals from other participants and thereby 
reducing noise. 

Having a single mixer for all participants, as taught in the PSTN art, would 
require an immense piece of software code i all of the variations in user 
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requirements were to be handled in a single 
software code would be unmanageably larg 
have simply not offered such diverse 

Similarly, the use of mixers 20, 22, 
5 network 18 offers a substantial improvemen 
teleconferencing as well. Typically, Internet 
ail participants, so each terminal receives 
participants. This places a tremendous 
connection to each user and a tremendous 

1 0 invention requires only the number of audio 
audio output, to be sent to the user. That is 
only one channel is required, and if stereo is 
sound, surround sound, central bass and 
corresponding numbers of audio channels 

1 5 required to each user and the loading on the 
Thirdly, it is also significant that the 
a modular manner. As will be described in 
software components of the invention are 
small modules designed to handle very 

20 than those like the existing PSTN. The moro 
addresses, the easier it is to design that 
of the software system. This is fundamental 
flexible and open. 

Other advantages of the invention wi 

25 description of the preferred embodiment 
example. 

Figure 2 presents an exemplary 
preferred embodiment of the invention, 
have direct access to an active network, whi 

30 term "active network* refers to a network tha 
software and other related software 

Participant A has a personal computer 
network 28 via a wireless connection 30 
serves Participant A is called a NetPort 32 



piece of software. As this piece of 
s, complex and slow, existing systems 



specific 



module 



having 



The 
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executing on the telecommunications 
over existing Internet based 
methods broadcast all voice streams to " 
to (N - 1) streams where there are N 

on the bandwidth of the final 
oad on the network, in contrast, the 
channels that the user requires at his 
if the user desires monophonic output, 
desired, two channels. Quadraphonic 
audio arrangements would require 
This greatly reduces the bandwidth 
network. 

nhixers of the invention are implemented in 
greater detail hereinafter, all of the 
implemented in small modules. Having 
tasks results in a far simpler system 
defined the task that a module 

and later, to integrate it into the rest 
to the provision of a system that is 



become more apparent from the 
whifch will be presented in terms of an 



physical 



layout of a teleconference in the 
four participants. Two participants 
a two are connected to the PSTN. The 
is operable to execute the mixer 
nts described hereinafter. 
(PC) 26 connected to a first active 
entity on the active network 28 which 
he specific role of the NetPort 32 within 



WO 00/19693 



10 



15 



20 



25 



30 




11 




PCT/CA99/00875 



the active network 28 will be described in greater detail hereinafter. The PC 26 is 
running a stereo-enabled Web browser with a RealAudio plugin that implements 
streaming audio and is output at speaker 34. The PC 26 also has a simple 
microphone plugin that passes samples fro n the on-board microphone 36 back to an 
IP address. Participant A also has a WebCam 38 connected to her PC 26. 

Participant B is connected directly tci a second active network 40, but in a 
location geographically remote from the first active network 28. Participant B also 
listens from a speaker 42 connected to his *C 44 through a streaming-audio 
application, but is talking through a wireless telephone 46 for its boom microphone. 
Participant B is plugged into a second NetPDrt 48 through a telephone jack and 
hardwired connection 60. 

Participant C is connected to the POTS 52 (plain old telephone system) via a 
plain black rotary-dial telephone 64. 

Participant D has two speaker phonos 54, 56 fed by two separate POTS lines. 
He also has an Internet connection via a PC 60 which runs a Web browser, but his 
Internet Service Provider (ISP) does not provide good enough quality of service 
(QoS) for voice, so he just uses it for the graphic user interface (GUI). 

A GUI is piece of software that prese nts data to users in a graphical manner, 
allowing for easy interpretation and modification. It is preferred that the invention be 
implemented in such a manner, where poss ble. The GUI runs as Java in the PC 
browser, and communicates with call proces sing applications running on the active 
network by means of sockets. Invoking it involves typing a URL (uniform resource 
locator such as l, coolPhones.com ,, ), after which it sits in a window waiting for an 
incoming call or a user input event to place a call. Inputs can be made via a mouse, 
keyboard, trackball, touchscreen, joystick or other similar manner. The GUI is strictly 
an interface, though, since it is unacceptable, for example, to have voice-mail fail 
when the PC is not active. Therefore, the real call processing decisions are made on 
the active network side. 

This exemplary system also includes an Internet network 62, which is 
connected to the PSTN using, for example, 
Protocol) connections. These connections ar 
Internet network 62 is also shown to be connected to both active networks 28, 40, but 
many other system topographies are also possible. The invention is not limited by 
any particular topography. 



H.323 and SIP (Session Initiation 
re known in the art, as are others. The 
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While the teleconference is in progress, Participant A hears Participant B on 
the left, Participant C in the middle and Part cipant D on the right, because Participant 
A is mixing the other participant's monophonic voice streams into a metaphorical 
stereo spectrum. The use of the stereo output has two major advantages. Firstly, it 
aids in identifying which participant is currently speaking. Secondly, it allows higher 
noise levels to be tolerable to participants due to the "cocktail party" effect. This 
effect acknowledges that people are able to 
an environment where there is considerable 



have a means for identifying and focussing t heir attention on a particular speaker. 
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converse comfortably with one another in 
background noise, provided that they 



The use of stereo sound has been shown to 
Stereo sound can be synthesized in 
implementation, amplifier gain can be varied 



provide this identification, 
a number of manners. In a simple 
between the left and right channels, for 



example: one participant may be played at full volume on the left channel and none at 



all on the right, a second may be played with 



15 none on the left, while a participant may be played at equal volume in both channels. 
More complex implementation of stereo may, for example, introduce a delay to the 
audio signal before playing it on one of the ciannels, simulating the additional time 
the sound takes to travel to the farther of the listener's two ears in a regular physical 
environment Such methods are generally kiown to those skilled in the art. 

20 Accordingly, Participant A's GUI sere an shows a Web page with Participant 

B's picture on the left, Participant C's number in the middle and Participant D's caller- 
ID name on the right. In each case, the GUI displays the best identification 
information that it has available for each part cipant. This identification may include a 
URL or telephone number. As well, "screen- pop" information could be provided 

25 which identifies personal information about the participant such as his address. 

As well, via the GUI, Participant A can click on u ear" and "eye" icons for each 
of the three participants on her screen to enable or disable their ability to hear her or 
to see her through the WebCam. Participant A can also drag on a "mouth" icon to set 
the volume level at which they speak to her, and drag participants left and right to 

30 adjust their stereo imaging. Alternatively, the volume level of participants could be 
governed by their physical location on the GUI display - the further a participant's icon 
is away from an icon representing the user, t ie lower their volume level. Another 
preferred feature of the GUI is that when someone speaks, their icon becomes 
brighter then gradually fades again with inact vity. 



full volume on the right channel and 
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Similarly, Participant B hears Partidsants A, D and C in a metaphorical stereo 
spectrum from left to right. Because he finds the sound of participant A's voice close 
to that of Participant C ( he has chosen to separate them spatially as much as 
possible. This is done with the same type c f GUI described with respect to 
5 Participant A above. 

Participant C hears a conventional f >ur-way conference call, with the voices " 
of the three other Participants companded smd mixed together. As a result, she has 
difficulty distinguishing Participants B and D. However, she has the flexibility to tailor 
the call to some extent with the preference i or single voice dominating, adding noise 

1 0 filters, or other functionality via her proxy. 1 his addition of functionality directly to 

PSTN customers is very significant. As explained in the background, PSTN services 
are driven by a supply model that only provides commodity services, and takes a 
long time to provide those limited services. There is a vast PSTN infrastructure which 
provides single monophonlc lines into millions of homes and businesses, all of which 

15 are shackled with these limitations. The use* of proxies in the manner of the invention 
provides greater flexibility and access to new services which may be implemented 
quickly and at very low cost. More details a b provided hereinafter regarding the 
preferred use of proxies. 

Participants who do not have the capability of interacting with the active 

20 network will have generic proxies assigned to them which are dictated by the nature 
of their telecommunication connections. For example, the system knows that a 
Participant only has PSTN access if that is t le connection the call manager has 
identified as the best connection during call setup. 

Participant D has a similar stereo arrangement, over which he has defined 

25 Participant A to reside on the left speaker te ephone, Participant D on the right 
speaker telephone and Participant C on both channels. This arrangement also 
creates a metaphorical stereo spectrum. Ot ler means are known in the art for 
carrying stereo over the PSTN, but such me hods generally require more complicated 
hardware at the Participant 1 s end. Participant D has the same GUI as Participants A 

30 and B so he is able to control his proxy and mixer on the active network directly. 

As an example, exemplary signal processing software for participant A is 
presented as a block diagram in Figure 3. Voice streams from other participants 
arrive in different forms and need to be conVBrted to a consistent form, companded 
and then mixed. They also need to be trans nitted across the radio link in an efficient 
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form, and other signal processing such as echo 
to be done on the voice data. In this case, RealAudio 
consistent form though MP3 or a number of otfter 
particularly convenient as it is a realtime streaming 
5 industry and for which many tools are currently 
This type of processing is also required for the 
different forms for each participant which corre 

Not shown in Figure 3 are such functiohs 
level control, though their implementation follows 

10 provided herein. 

Specifically, pulse code modulated 
Participants C and D, which are connected to 
is the standard transmission form for audio in 
received from Participant B is already in RealAudio 

1 5 packets and is easily transported over IP, it is 
passing its signal to the RealAudio mixer 68. 

The RealAudio mixer 68 combines the 
with the participant's requirements. In Figure 
line is shown which interfaces the RealAudio mjxer 

20 card 72. The audio output of the RealAudio mifcer 
well, and also to a PCM converter 74. 

This PCM converter 74 feeds the echo 
more or less matches the output from the participant 
echo cancellor 76 can remove the speaker output 

25 up by the microphone 36. The PCM signal 
converted to RealAudio at the voice coder 78, 

RealAudio packets are numbered sequentially 
arranged in the proper order when they are deqoded 
to time stamp packets as the time delays are 

30 packets that result from their transport from different 
will not generally be detectable by the participants 
deliberately add a delay to the incoming signals 
signal jitter. As the data is arriving in finite and 
inevitably be some degree of jitter, so buffering 



voice streams are received from 
RealAudio converters 64 and 66. PCM 
PSTN. Since the voice signal 

format, which comprises data 
necessary to convert it before 
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cancellation/suppression may need 
has been chosen as the 
forms qpuld be used. RealAudio is 
standard that is well known in the 
available, such as codecs and mixers, 
other listeners, and it takes slightly 
ate to their respective setups, 
as encryption, tone controls and 
logically from the description 



incoming audio signals in accordance 
, a bi-directional Activity and Control 
68 with the PC 26 via an Ethernet 
68 goes to the Ethernet card 72 as 

^ancellor 76 with an audio signal that 
;'s speaker 34. This way, the 
signal that is inadvertently picked 
the echo cancellor 76 is 



to ensure that they are 
Generally, it is not necessary 
, and the varied delays in data 

sources, or by different routes, 
. In fact, RealAudio may 
by storing them in a buffer to absorb 
distinct data packets, there will 
is preferred. A buffer that causes 20 
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bsorb the effects of jitter most of the time, 



10 



15 



20 



25 



30 



mSec - 50 mSec delay is sufficient time to at 
and is not long enough to annoy the users to a great extent. 

Also, note that all audio signals passing between the NetPort and NetPort 
Manager are in RealAudio form, and not PCM. This makes the transport over the 
digital interconnection more convenient. 

Other preferred aspects of the teleconference are outlined as follows: 

1. Call Setup 

The most important aspect of call se up is the identification of the participants, 
where they can be found and then of course, creating the connections. In the 
preferred embodiment, the teleconference will be created by one or more of 
the participants who are GUI-enabled. These participants will advise the 
network of the identities of the partici pants and the call setup software on the 
network will make the connection wit i the participants. 
Some of the participants will have internet addresses, while others will have 
telephone numbers. In each case, tie call setup software will investigate the 
participant and establish the best po; sible connection that it is aware of. 
Those participants without Internet aixess will be assigned proxies which 
reflect the resources they have access to. For example, if the call setup 
software identifies a PSTN telephone number as the best connection, it will 
assign a PSTN proxy to that participant unless advised otherwise. 
In the preferred embodiment, all part cipants who are GUI-enabled can add 
participants to the call, but for high-security teleconferences, addition of 
participants should be controlled by s single participant 

2. Telecommunications Operating System 

The telecommunications operating system aspect of the invention provides 
unified control and access to all system resources and networking links, with 
the functionality in and implied by Fig ure 3. This represents a large collection 
of signal processing and control fund ions connected together in response to 



the commands of the callers. 
This contrasts with the "pure Internet 
cooperating tasks in all of the various 



makes it very difficult to optimize and 



Protocol 0 approach which require 
computers to arrange to do their parts 



of the processing through an applicat on-specific protocol built on socket() 
calls, with no single program having in overview of the whole setup. This 



manage the system, and each such 
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application has to reinvent call process 
part of the underlying implementation 
invention is built as a middleware layer 
In "telephony classic 0 one would not 
generality in software, but would make 
hardware that assumes all inputs and c 
numbers to call in order to connect to it 
makes this a "closed system 0 , in which 
limitations on who can develop new 
Proxy 

It is preferred that the invention be 
graph model as described in the 
Patent Cooperation Treaty, Serial No 
for Telecommunications". While a 
and manage the entire connection, 
well as voice streams, that application 
"agents" from "proxies". 
A proxy is a piece of software that acts 
connection. In this case Participants A 
and Internet providers are separate 
contains data that represents the 
whether Participant A is already on the 
network's 28 voice trunk is getting full, 
do specific tasks, such as responding tc 
managing a voice call in progress. 
The terms proxy and agent are 
For the purposes of this document, they 
agents, each of which handles a specia 
not comprise an immense block of code 
in its simplest form, is merely a 
as required, discarding them when their 
These agents are sent to parts of the 
going on and are connected to the signal 
through a "controlling application". This 



ng. The invention uses socket() as 
irt a preferred embodiment, so that the 
on top of IP. 
attempt to set up something with this 
special "stereo conferencing server" 
utputs are PCM and would add speciaf 
The need for specialized hardware 
nnovation is slowed down by 
telecommunications applications 

applied to a network which employs a 
co-per|ding patent application under the 
_J titled "Connection Manager 



single application program is used to see 
including IP and videoCam connections as 
constructed by collecting together 



sometimes 
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on behalf of a specific party to a 
hrough D, and each of the networks 
parses with separate proxies. A proxy 
and state of the party, such as 
telephone and whether the first active 
has components that are agents to 
off-hook on a telephone and 



preferences \ 



end I 



used interchangeably in the art. 
are distinct: a proxy is built out of 
situation. Therefore, the proxy does 
with all conceivable functionality, but 
or which instantiates software agents 
tasks have been completed, 
system in which signal processing is 
processing code or hardware 
architecture of proxies, agents and 
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controlling applications is what 
appreciate the whole structure of a 
several different parties 
Proxies should persist in the 
5 example, a user's forwarding 

preferred that persistence be 
continuously updated, so that all 
the communication. In the event of 
around the failure, allowing the 

10 transactional interaction techniques 

In the "pure Internet Protocpr 
on hardware belonging to the 
through socket() mechanisms in an 
additional structure on top of this. 

15 In "telephony classic" there is a si 

database for all users and decides 
is too large to modify quickly, and 
manufacturer. Again, this approach 
of new features. The architecture 

20 understand and modify software, 

system to be open to software 
brought to market very quickly. 
4. Graph 

It is preferred that the invention be 

25 graph model as described in the 

Patent Cooperation Treaty, Serial 
Configuring Communications 
the signal processing and 
which is later implemented by taking 

30 libraries and modifying them, either 

the IP addresses to which they makje 
interconnection structure specified 
communication among the agents, 



i Systems' 



I communications 
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connection management applications to 
connection while still being "owned" by 

preserice pf component failures, so that, for 
instructions do not get lost during a crash. It is 
provided via a distributed database which is 
concerned parties are aware of the status of 
a failure, the system is able to work 
communication to continue. Such 

are known in the art. 
approach there is only custom software running 
various parties involved. and communicating 
ad hoc protocol. The invention builds an 

sindjle very large program that looks at a 
what they would want to do. This program 
only be modified by the equipment 
is not flexible enough for rapid evolution 
the invention makes it easier to 

the same complexity, allowing the 
development, so that new features may be 



can 



of 



without 1 



npplied to a network which employs a 
cojpending. patent application under the 

_ , titled "Method and System for 



No 



Briefly, the graph model constructs 
structure as a mathematical graph, 
filters" that implement the nodes out of 
by a dynamic linking process or by setting 
socket connections, to have the 
the edges. This graph is also used for 
the data structure that defines a 



by 



as. 



10 



15 



20 



25 



WO 00/19693 



I 



30 



-18- 



connection. An API layer that describe s characteristics of these graphs is 

added above the raw graph structure to assist in writing agents. 

In addition to filters, it is also preferred that this graph data packet contain 



calls to proxy agents required to set up 



connection and negotiation processes, 
standard API simplifies the creation of 



a very logical and understandable form 
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the call. Proxies may also send their 



agents to collaborate on building and managing graphs. 
An application programming interface (API) converts a series of comparatively 
simple and high level functions into the lower level instructions necessary to 
execute those functions, simplifying uss of an operating system. Using 
Windows APIs, for example, a program can open windows, files, and 
message boxes, as well as perform me re complicated tasks, by executing 
single instructions. 

The particulars of how an API for the invention is implemented are not critical, 
but it is desirable that a standard API be employed that expresses control, 



including payment. The use of a 
lew features by third parties. 



A GUI is particularly well suited to the use of a graph model, as the GUI may 
present the assembled filters as defined by a graph data packet, to the user in 



It is also preferred that the GUI have 



the functionality to let the user modify tpe graph data packet simply by altering 
filters and their interconnections. 
In the "pure Internet Protocol" approach the overall communications structure 
is not visible at all, while in the "telephony classic" approach it is possible for 
switch software to connect physical ports together, but not to pull functions out 
of libraries. The decision about what ports to connect together is explicitly 
made by the users by dialling telephone numbers in the "telephony classic" 
approach. 

Real Time Operating System (RTOSl 

Voice teleconferencing is a real time pneedure, so RTOS should be used as 
known in the art. Generally, RTOS's divide code to be executed into smaller 
units of threads and functions, and then schedule the execution of these 
threads and functions to be performed Drior to specific deadlines. 
Distributed Operating System 
A distributed operating system is one iii which portions of the software can run 
on different nodes. In the case of a telecommunications system, distribution 
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of software makes it easier to maint ain real time operation as there are more 
options available to schedule timely execution. As well, distributed operation 
improves scalability and speed. Tho use of agents and proxies lends itself to 
the efficient use of a distributed system, in that agents and proxies may be 
assigned to run on different nodes of the system. Ideally, agents will be 
located close to where they are required, to minimize time delays in 
communicating with the entities they represent. Such a suitable distributed 
RTOS is described in the co-pendin j patent application under the Patent 



Cooperation Treaty, Serial No.. 



User processes running as "filters", 
come from being in the signal path, 



..titled "Distributed Telco" 



Figure 4 presents a block diagram of an exemplary operating system 
architecture in a preferred embodiment of the invention. A distributed 
communications substrate 80 is interposed between user processes and the 
underlying machines, so that processes can generally be moved from one 
machine to another without being av/are of it, either to distribute load or to 
recover from failures. 

Processes running in the system cone from different sources and accordingly 
get different treatment in terms of the trade-off between security and 
performance. Call-processing functions acting on behalf of the end users run 
in a protected "sandbox" environment on a virtual machine. Those working 
on behalf of the network provider mny run there, but may also be 
implemented directly as processes running on the network operating system. 

with the hard real-time demands that 
and also run directly on the 
communications; substrate 80. Processes belonging to different users are 
protected from each other by the usual operating system mechanisms such 
as memory mapping and file privileges, but the source is also reviewed by the 
network administrator. Filter processes on the same machine and part of the 
same call may share an address space and a thread of control, with data 
being passed with a function call mechanism and with connections to other 
hardware being handled by a stub that adapts a function call to a socket-type 
mechanism. These filters would still be dynamically linked, even with the 
function-call mechanism. 

Signals pass through filter processes F, which also implement drivers and 
performance-sensitive functions on behalf of the network. Call processing on 
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behalf of users is handled by CP 
machine VM environment, which 
can transfer control on failure to a " 
run on the common software 
. 5 appropriate physical systems and 

processes S also run on the 
the hard real-time constraints of the 
functions are one type of server 

5. Mapper 
1 0 The allocation of taste and network 

graphs is done by an optimizer called 
many ways that a graph representing 
allocated to a physical network: each 
nodes, for example, and sometimes 

15 which data can be carried. The si 

proxies about where to put radio link£ 
applies a "greedy algorithm 0 to put 
work edge as possible. A good 
which local decisions are made with 

20 possible. It is not essential to get a 

not seriously wasted. 
In the case of the invention, the mos 
problem is the handling of the voice 
each of N participants, each mixer w 

25 mapper must balance the benefit of 

processors against the extra cost of 
The factor that governs these decision 
is quality of service (QoS). 
known in the art, but in the case of 

30 each set of participants and network 

6. Negotiation 
It is preferred that the architecture for 
system described in the co-pending 
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running in a secure virtual 
includes checkpointing functions that 
'dhost machine" 82. All these processes 
communications layer 80, which places them on 

arranges for their connections. Server 
commu nications substrate 80, but do not have ~ 
Filter processes. Secure call-processing 
process. 

I capacity to different communications 
the mapper. There are in general 
a desired communication can be 
of the fitters can run on several different 
here are several types of links over 
embodiment uses hints from the 
, that is, after the voice coder, and then 
computing resources as close to the net- 
should be a distributed application in 
ocal knowledge to the greatest extent 
global optimum, as long as resources are 

significant resource management 
^treams. In having a separate mixer for 

receive (N-1) voice streams. The 
distributing the mixers among various 
ransporting redundant audio signals, 
in the implementation of the invention 
Methods bf distributing such real time loads are 
f thje invention, the solution will vary with 
opography. 

agents provide for use of the negotiation 
riatent application under the Patent 
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Cooperation Treaty, Serial No., 



tie 



the 

Cay, 



i wit i 



Negotiating Telecommunication 
Many users are competing at once 
(including its computing capacity), 
resolve thjs contention: agents for 
payment as part of connection setub 
all parties have accepted it. A callejr 
reduced quality level if the cost of 
high. For example, on Christmas 
users can expect to get through 
a busy signal. 

In "pure Internet Protocol", temporary 
efforts" basis and packets may be 
longer time-scale by overprovisioniihg 
frequent. With differential service a 
defined, but the definition needs to 
invention can be used to manage 
access in such a way as to permit 
In "telephony classic" contention is 
first served) and again the network 
too frequent 

Negotiation management may be 
for each of the user terminals and 
networks. Each negotiation agents 
telecommunications system, for 
represents the interests of its 
over the telecommunications network 
the negotiation and then passing a 
proposed connection, to each 
negotiation agent may either accepjt 
proposal to the other negotiation 
When all or part of the graph data 
assembles the listed filters in the 



Resources. 
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titled "Method and System for 



r or the shared resources of the network 
t is preferred to apply a market model to 

parties involved offer and demand 
, and a connection does hot happen until 
can choose to try a connection at a 
high quality connection becomes too 
, the load may be temporarily high, so 
reduced voice quality rather than getting 



congestion is resolved on a "best 
almost arbitrarily thrown out, and at a 
the network so that failures are not too 
small number of priority classes are 
be managed. The market model of the 
dfferential service, allocating high priority 
guarantees on service, 
managed by call admission (first-come, 
is overproyisioned so that failures are not 



fDM 



implemented by having a negotiation agent 
each of the multiple telecommunications 
is operable to execute somewhere on the 
example, on the active network, and 
respective party in negotiating communication 
This is done by identify participants in 
graph data packet which describe the 
participant for their consideration. Each 
, reject or revise it to make a new 
agents. 



jacket is to be executed, a device simply 
Planner defined in the graph data packet. 
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It is also preferred that the invention be 
mechanism that protects proxies from erroneous 
As well, it is presently desirable that proxies 
another language with similar advantages 
. 5 include: 

a. excellent security 

b. a large community of experienced 

c. object oriented code structure 

d. simple net-based distribution 
10 A telecommunications system implemented 

above provides a foundation for the mixed meqia 
greater flexibility and power to existing service^ 
and Internet gaming. 

Other options for implementation of the 

15 1. Companding 

Companding techniques use "compression 1 
(smoothly) so as to keep a signal's levejl 
algorithms that adjust gains to 
techniques used in audio are frequency 

20 companding which adjusts filter cutoffs 

signal levels are low. 
An extreme example of expansion is "j 
level below a certain threshold are turned 
noise. In telephony the most common 

25 opposed to "cancellation", in which the 

its gain reduced, which reduces the looja 
oscillation. Companders use around 
Instantaneous companders work on a 
common A-taw case is covered under 

30 2. Voice coding 

Voice coders are used to reduce the 
signals. There are many types, but 
minimizing some mathematical measurfe 
source; or they can model what the ear 
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implemehted-with a strong security 
or malicious code in other proxies, 
agents be written in Java™, but 
also be used. Advantages of Java™ 



developers 

mechanism 

with the functionality described 
applications of the future, and for 
such as high bandwidth telephone, 

invention include: 

algorithms that try to adjust gains 
more constant and "expansion" 
te signal-level variations. Some 
dependent, such as Dolby 
to suppress background hiss when 



sic 



5-50i 



squelch" in which signals with power 
off completely to minimize idling 
variant is "echo suppression", as 
gnal path from the quieter user has 
gain for echoing and feedback 
operations per sample, 
dample-by-sample basis, and the 
:oders" below. 

bgndwidth requirements for voice 
brojadly they can apt on the waveform, 
like error power; they can model the 
will notice. Coding for compression is 
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i variant 



an active research area, and a 
appear. 

'Telephony classic" uses waveform 
law). Sampling is done at 8kHz on 
300Hz to 3300Hz. The passband 
on speech quality and intelligibility, 
33% margin over the Nyquist rninii 
prefilter costs. A-law and p-law are 
representations, chosen as a way 
a wide range of signal levels. By 
stereo 16-bit fixed-point sampled at 
the bandwidth and the use of a T1 
sample to sample, the same quality 
with ADPCM (adaptive differential 
speaking, digitizes the derivative 
Most digital cell-phones use a 
model the incoming sound in terms 
cords or airflow and which in turn 
requires less bandwidth than 
more slowly than the waveform, bu 
speech or even for speech in a 
are an active topic of research and 
output rates anywhere from 4kb/s 
SOMIPs of DSP capacity. Coders 
and hence add at least that much 
Source coders typically try to detecjt 
silence typically. saves about 50% 
side it is conventional to replace sil 
listeners know the connection is sti 
Source coding is difficult to use for 
model a large number of different 
early digital audio such as CD and 
enough bandwidth and dynamic 
ear. Minidisc and digital compact 



steady stream of new coders is likely to 



coding in the form of 8kHz A-law (or m- 
a signal filtered to pass the range from 

defined to get good subjective scores 
and the sample rate is designed with a 
in a trade-off between network and 
specialized 8-bit floating-point 
get roughly constant signal-to-noise over 
comparison, compact disc (CD) sound is 
44.1kHz, which requires roughly 24 times 
ine. Because speech varies slowly from 
can be had for roughly half the bandwidth 
dulse-code modulation) which, roughly 
instead. 
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of linear prediction coding, which tries to 
of a sound source that simulates the vocal 
drives a filter that models the larynx. This 
coding because the larynx moves 
works badly for anything other than 
environment. These "source coders" 
currently produce tolerable speech at 
A typical modern coder uses about 
typically operate on 20msec frames of data, 
to the signal path, 
silence, and avoiding the transmission of 
bandwidth on average. At the decoding 
silence with "comfort noise" so that the 
I live. ' 

music, because it would be necessary to 
instruments alone and in combination, so 
DAT, just used waveform coding with 
range to satisfy (more or less) the human 
cassettes brought in coding that reduced 
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CD bandwidth by a factor of about 10 b/ using psychoacoustics. 
Psychoacoustics applies, in particular, masking effects, where loud tones 
mask nearby ones for normal ears, arte 
transmitting the inaudible components, 
rate-adapted, as in RealAudio, and is a 



speech applications in the system of thu invention, 



Conventional filters, companders and s 



bandwidth can be saved by not 
This type of technique can also be 
good candidate for high-quality 



milar components will not work on a 



coded signal, so it is standard to decompress before filtering. In some cases 
this may be avoided, for example, N-wsiy combining can take advantage of 
silence to do companding at no additional cost of bandwidth, and only needs 
to decode and recede during bi-directional conversation. 
MPEG (Motion Picture Experts Group) 

MPEG coders do the same type of thinp for video signals that perceptual 
coders do for music. Components of a 

frequencies are digitized at low resolution, using 8*8 discrete cosine 
transforms to do the filtering, and using "motion estimation 11 so that 
components of an image that can be dorived from adjacent frames are not 
retransmitted. MPEG decoding is prefe rably left for the end-user's PC, 
because It is very demanding and beca jse specialized hardware exists for it. 
However, the traffic properties are an inportant consideration in implementing 
the invention. Straight digitized television requires roughly 30 frames / sec * 



colour, for 144Mb/s. That is beyond 
MPEG2 gives similar quality at 2Mb/s; 
MPEG2 is also bursty, needing more 



of msec of delay and the picture is 



200 kpixels / frame * 3 colours * 8 bits / 
what 3G wireless is built to handle, but 
hence the 3G requirement for that rate, 
capacity when the image changes suddenly. 
At the low-quality end, videoconferencing is usually done at 128kb/s. At this 
rate the coding process adds hundreds 
poor. 

If there is high demand for full-motion video, then 5MHz slots will not have 
sufficient capacity, but 20MHz slots ancjl generous use of antenna diversity 
could support 10-40 users at that rate 
The network operating system could initiate processes in the end-user's PC 
so that video services can be set up eajsily 
Other applications 
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Other applications such as animated 
locations, voice activation, automat 
signal shaping to compensate for 
or software in the system, are all 
5 invention. 

Examples have been shown to 
but the number of variations is by no mear 
could be made for any telephony device, 
machines, pagers, point of sale computers], 
10 private branch exchanges. While particular 
have been shown and described, it is clear 
made to such embodiments without departing 
invention. 

The invention could also be 
1 5 Internet and PSTN networks. For exampty 
the functionality of the invention similar to 
PSTN a specialized server could be attached 
implementations would not have all the 
certain aspects of its teachings. 
20 The method steps of the invention 

machine code stored in a variety of formatjs 
Such code is described generically herein 
program for simplification. Clearly, the 
with the code of other programs, implemented 
25 calls or by other techniques as known in 
The embodiments of the invention 
or similar device programmed in the manner 
an electronic system which is provided wi 
Similarly, an electronic memory means 
30 Access Memory (RAM), Read Only Memo|ry 
storage media known in the art, may be 
As well, electronic signals representing th£se 
via a communication network. 



video, stereo input at the participant's 
c gain control (AGC) at the user's PC, and 
frequency response of certain devices 
Mown in the art, and easily applied to the 



i the i 



I witi 
su :h 



4 



PCT/CA99/00875 



demonstrate various aspects of the invention, ~ 
s complete. Comparable implementations 
luding personal digital assistants, fax 
amateur radios, local area networks or 
embodiments of the present invention 
that changes and modifications may be 
from the true scope and spirit of the 

implerrjented to a lesser extent on existing 

, Internet servers could be given much of 
; applications such as NetMeeting. On the 
to a class 5 switch. These 
benefits of the invention, but could apply 

nay be embodied in sets of executable 
such as object code or source code, 
as programming code, or a computer 
executable machine code may be integrated 
as subroutines, by external program 
art. 

may be executed by a computer processor 
of method steps, or may be executed by 
means for executing these steps, 
computer diskettes, CD-Roms, Random 
(ROM) or similar computer software 
programmed to execute such method steps, 
method steps may also be transmitted 
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It would also be clear to one skilled in the art that this invention need not be 
limited to the described scope of computers and computer systems. The principles of 
the invention could be applied to citizen's band radio, amateur radio, or packet radio. 
Again, such implementations would be clear t<|> one skilled in the art, and do not take 
away from the invention. 



